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Abstract: We investigate the performance of a family of multiple com- 
parison procedures for strong control of the False Discovery Rate (FDR). 
The FDR is the expected False Discovery Proportion (FDP), that is, the 
expected fraction of false rejections among all rejected hypotheses. A num- 
ber of refinements to the original Bcnjamini-Hochberg procedure [1] have 
been proposed, to increase power by estimating the proportion of true null 
hypotheses, either implicitly, leading to one-stage adaptive procedures [ i, 7] 
or explicitly, leading to two-stage adaptive (or plug-in) procedures [2, 21]. 

We use a variant of the stochastic process approach proposed by Gen- 
ovese and Wasserman [11] to study the fluctuations of the FDP achieved 
with each of these procedures around its expectation, for independent tested 
hypotheses. 

We introduce a framework for the derivation of generic Central Limit 
Theorems for the FDP of these procedures, characterizing the associated 
regularity conditions, and comparing the asymptotic power of the various 
procedures. We interpret recently proposed one-stage adaptive procedures 
[1, 7] as flxed points in the iteration of well known two-stage adaptive 
procedures [2, 2i]. 
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1. Introduction 

Multiple testing problems arise when many binary tests are performed simulta- 
neously. The rejection of all hypotheses with individual p-values smaller than a 
fixed threshold results in an increasing number of false discoveries as the number 
of tests increases. There is therefore a need for a risk measure taking multiple 
testing into account. Multiple testing procedures use a collection of p- values as 
input and output a set of hypotheses to be rejected. 

Since the seminal article by Benjamini and Hochberg [1]. the False Discovery 
Rate (FDR) has been accepted as a practical and convenient method for risk 
assessment in multiple testing problems involving high-dimensional data analy- 
sis, including non parametric estimation by wavelet methods in image analysis, 
functional magnetic resonance imaging (flVIRI) in medicine, source detection in 
astronomy, or DNA microarray analysis in biology. The FDR is the expected 
proportion of erroneous rejections among all rejections. 
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The procedure originally proposed by Bcnjamini and Hochberg [1], which we 
term procedure BH95, controls FDR when the true null hypotheses are inde- 
pendent, or display certain forms of positive dependence [1, 3]. Considerable 
efforts have been devoted to the development of procedures retaining the FDR- 
controlling capabilities of the BH95 procedure under more general conditions of 
dependence [3, 19], and/or with higher power [2, 4, 7, 21, 23]. The second aim 
is driven by the observation that the original BH95 procedure actually controls 
FDR at level of exactly 7roa, where ttq is the (unknown) proportion of true null 
hypotheses [3, 8, 19, 23]. When ttq < 1, applying the BH95 procedure at level 
a/TTo would increase the number of rejections while keeping FDR < a. However, 
as TTo is unknown, this is only an Oracle procedure. Many proposed procedures 
therefore try to imitate the Oracle by applying the BH95 procedure at level 
q/tto, where ttq estimates (or at least provides an upper bound for) the true 
TTo [2, 21, 23]. Such procedures are referred to as two-stage adaptive or plug-in 
procedures. A new class of procedures has recently been proposed, with the aim 
of providing tighter FDR control than the BH95 procedure, while avoiding ex- 
plicit solution of the semi-parametric problem of ttq estimation. Procedures of 
this second class are referred to as one-stage adaptive procedures [4, 7]. 

The FDR controlling properties of such procedures have been carefully studied 
for a finite number of hypotheses hypotheses [1, 2, 4, 8, 9, 19, 21], or asymp- 
totically [5, 7, 8, 10, 11, 21, 22, 23]. As the proportion of erroneous rejections 
(FDP) is a stochastic quantity, its fluctuations around its mean value are worth 
investigating. Several procedures have been proposed for controlling the upper 
quantiles of the FDP [11, 12, 13, 15, 16, 17, 18, 26]. The asymptotic behavior 
of process (FDPm(t))o<t<i, where i is a deterministic threshold, has also been 
studied [11, 21, 23]. We focus here on the properties of the random threshold r 
associated with a given multiple testing procedure, particularly in the asymp- 
totic distribution of FDP,„(r), the FDP actually reached by the procedure. 

Organization of the paper. In section 2 we propose a general framework 
for asymptotic analysis of the FDP of multiple testing procedures. In section 3 
we derive the asymptotic distribution of the FDP of a multiple testing procedure 
with generic threshold function T and characterize the asymptotic equivalence 
of multiple testing procedures. These results are expHcitly connected to the 
regularity of the map T, which is then discussed. In section 4 we derive the 
asymptotic behavior of several existing procedures. In section 5 we point out 
interesting connections between one-stage adaptive and two-stage adaptive pro- 
cedures. The main results are summarized and discussed in section 6, and proofs 
of the main results are gathered in section 7. 

2. Background and notation 
2. 1 . Background 

Throughout the paper, we consider a sequence (Pi)igN of p-values associated 
with a collection of binary tests of a null hypothesis TYq against an alternative 
hypothesis Tii. 
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Definition 2.1 (Multiple Testing Procedure (MTP)). A multiple testing pro- 
cedure Ai is a sequence of functions Aim '■ [0, 1]™ —f [0, 1] such that for any 
m- dimensional vector of p-values (Pi, . . .Pm), hypotheses i satisfying 

P, < Mm{Pu . . . P,n) ■ 

are rejected. Slightly abusing notation, we shall write Ai{Pi, . . . Pm) for 

MmiPl,.--Pm)- 

Denoting by Vm and Rm the number of illegitimate rejections and the total 
number of rejections among the m tested hypotheses for a multiple testing 
procedure A4, the associated False Discovery Proportion and False Discovery 
Rate are FDP™(A^) = j^Tn^ and FDR„(A^) E -j^^ . 

Model. We consider the setting originally proposed by [1]: among m tested 
hypotheses, mo (to) are true nulls. We assume that p-values are uniformly dis- 
tributed on [0,1] under Tio, and distributed according to Gi under Tii, where 
Gi is a concave, G^ distribution function, with density gi. 

We assume that 7TQ{m) = mo{m) / m verifies limm^+oo ttqCto) = ttq, which we 
call proportion of true null hypotheses. We let G{x) = ttqx -I- (1 — 7ro)Gi(a;), and 
= ttq -|- (1 — 7ro)5i be the corresponding density. Finally, we assume that all 
p-values are independent. 

The BH95 procedure. Letting (P(i))i<i<m be the vector of ordered p-values 
associated with {Pi)i<i<m, and 

Im ~ max {i G {1, . • . to}, P(,;) < ai/m^ , 

the BH95 procedure at level a is defined by A^"j(Pi, . . .Pm) = ctlm/m- This 
definition can be rewritten as follows. Letting be the empirical distribution 
function of the p-values, 

X"(Pi,...P,„) = sup{ue [Q,l],Gm{u)>u/a} . 

These two equivalent formulations of the BH95 threshold are illustrated in Fig- 
ure 1. Following [7], u i— > u/a will be called the rejection curve of the BH95 
procedure (also known as Simes' line [20]). 

2. 2. Formalism 

Threshold functions. This interpretation of the BH95 procedure in terms 
of the empirical distribution function suggests to define threshold functions as 
follows. Let I?[0, 1] denote the set of cadlag functions defined on [0,1], that 
is, the set of functions defined on [0, 1] that are right-continuous and have left 
limits. 
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Figure 1. Dual interpretations of the BH95 threshold. 



Definition 2.2 (Threshold function). A multiple testing "procedure Ai has thresh- 
old function T : D[0, 1] — > [0, 1] if and only if 

VmeN,Al(Pi,...P,„) =r(G™). 

Note that T does not depend on m in Definition 2.2. In the remainder of this 
paper T(G) will be denoted by r*. 

FDP as a stochastic process of a random threshold. As suggested in a 
previous study [il], the False Discovery Proportion can be viewed as a stochastic 
process. Let Go,m and <Gi,m denote the (unobservable) empirical distribution 
function of the p-valucs under the null and alternative hypotheses: 

mo(m) 

^ true} ^P.<' ■ 

m— mo(r/i) 

As 7ro(m) is deterministic in our setting and verifies lim,„^_|_oo TToim) = ttq, we 
will assume without loss of generality that it is constant equal to ttq, in order to 
alleviate notation. Therefore, we have Gm = 7roGo,rn + (l — 7i'o)G'i,,ii, and, for any 
t GjO, 1], i?„,(t) 1 lp.<t = G™(t) and V^{t) = E{V«„ true} lp.<t = 

7roG-o,m(Oi so that 

FDP„(t) = ^"^".™(*) 




is the False Discovery Proportion achieved at the deterministic threshold t. The 
asymptotic properties of the stochastic process (FDPm(i))Q<j<-^ were analyzed 
by Genovese and Wasserman [11]. They noticed that FDRm{t) = E[FDR,„(t)], 
so the achieved FDR at t, may be written as 

FDR,„(i) = pW(l-(l-G(Or), 
where p{t) = is the positive False Discovery Rate (pFDR) at t, as defined 
by [21]. They proved that the FDPm process converges to pFDR at a rate 
and built confidence envelopes for the FDP process using this result. 
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We make use of this stochastic process approach here to study the behavior 
of the FDP actually achieved by a given multiple testing procedure T , that is, 
the random variable FDP,„(T(Gm))- We investigated the asymptotic behavior 
of this variable and, in particular, its fluctuations around the asymptotic FDR 
achieved by procedure T, by writing FDRn^(T((G„i)) as a function of the empir- 
ical distribution functions under the null and alternative hypotheses. Letting 

V : (^^0, F^) ^ 7roFo(r(7roFo + (1 - 7ro)Fi)) 

and 

n-.F^ F{T{F)), 
the FDP achieved by procedure T may be written as 



FDP„(r(G„)) 



"^(ttoGo,™ + (1 - 7ro)Gi,,„) V 



since G„i = ttoGo,™ + (1 ^ 7ro)Gi^,„. Using the functional Delta method [27], this 
formalism makes it possible to break down the analysis of FDPm(T((Gm)) into 
the regularity properties of the map T, which depend solely on the procedure, 
and the asymptotic behavior of the empirical distribution functions of the p- 
values, which can be derived from Donsker's invariance principle [G] because 
p- values are assumed to be independent. 

Remark 2.3. This paper focuses on FDP, but the formalism we propose here 
could be used to derive the asymptotic distribution of any risk measure based 
on the number of true/false positive/negatives, under the same regularity con- 
ditions. In particular, the results obtained here can also be applied to the False 
Non-discovery Proportion (FNP) [10]: 

P^p^^(^) ^ {l-Rra{t)/m)-{^o-V,^{t)/m) 

1 - TTo 

Multiple testing procedures studied. The threshold function of the BH95 
procedure is defined by 

r{F) = sup{m G [0, 1], > u/a} . 

As the BH95 procedure keeps the false discovery rate at a level of (exactly) n^a 
when p- values are independent [3, 8, 19, 23], it is conservative by a factor ttq. 
Other multiple testing procedures have been proposed that estimate ttq, either 
implicitly or explicitly, to provide tighter (i.e. more powerful) FDR control under 
independence: 

One-stage adaptive procedures (BROS ['], FDR08 [7]) use rejection curves 
other than Simes' line, without explicitly incorporating an estimate of ttq. 

Two-stage adaptive procedures (BKY06 [ ;],STS04 Sto02 [21]) apply 
the BH95 procedure at a level of ck/tto, where ttq is an estimator of ttq. 
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We therefore consider threshold functions of the form 

T(F) =U{F,A{F)), 

with 

U{F,a) :=sup{ue [0,1], > r„(u)}, 

where : [0,1] IR.+ will be called a rejection curve (after [7]), and A : 
-D[0, 1] [0,1] will be called a level function, r^ will be denoted by r(a, •) 
whenever the dependence on a is of importance. A and arc two degrees of 
freedom that can be used to describe generalizations of the BH95 procedure, 
corresponding to the case in which the level function is constant (equal to a), 
and the rejection curve is Simes' line. In this paper we consider increasing 
rejection curves satisfying ra{0) = 0, so that U{F,a) > for any F e £'[0, 1] 
and a £ [0, 1]. 

2. 3. Overview of main results 

Theorem 3.2 below shows that the FDP of a multiple testing procedure with 
threshold function T converges in distribution at rate Xj ^fm to a conservative, 
procedure-specific FDR level. This theorem holds under a general regularity 
condition on the map T, which is implied by the existence and uniqueness 
of an interior right-crossing point between the distribution function of the p- 
values and the rejection curve of the procedure; the existence condition for a 
given procedure may be interpreted as a natural generalization of the notion of 
criticality, which has recently been introduced for the BH95 procedure [o]. 

Although the BH95 procedure is known to control FDR at a level of exactly 
■KoOL [3, 8], other procedures have been proved to yield only an FDR not larger 
than a, either for a finite number of hypotheses (procedures STS04, BKY06 and 
BROS) or asymptotically (Sto02 and FDR08). In section 4 we derive the asymp- 
totic behavior of these procedures, and the associated regularity conditions. As 
all procedures converge at the same rate 1/^/m, their asymptotic power may be 
explicitly compared through their attained asymptotic FDR. 

In section 5 we demonstrate the existence of interesting connections between 
the one-stage and two-stage adaptive procedures under investigation: with a 
striking symmetry, procedure BROS may be interpreted as a fixed point of the 
iteration of procedure BKY06, and procedure FDROS as a fixed point of the 
iteration of procedure Sto02. 

3. Asymptotic properties of threshold procedures 

This section provides general results about multiple testing procedures with 
threshold functions satisfying the following regularity condition. We refer to 
[27] for a formal definition of Hadamard differentiability. 
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Condition C.l (Hadamard-differcntiability). The threshold function T sat- 
isfies T{G) > 0, and is Hadamard-differentiable at G, tangentially to C[0, 1], 
where C[0, 1] is the set of continuous functions on [0, 1] The threshold function 
derivative is denoted by Tq. 

We begin by deriving the asymptotic distribution of the FDP of any mul- 
tiple testing procedure satisfying Condition C.l (section 3.1). We then define 
and characterize asymptotic equivalence between multiple testing procedures 
in terms of Condition C.l (section 3.2). Finally we interpret this Condition in 
terms of crossing points between the distribution function G of the p- values and 
the rejection curve (section 3.3). 



3.1. Asymptotic False Discovery Proportion 

Condition C.l makes it possible to use the functional Delta method [-7] to derive 
the asymptotic distribution of the False Discovery Proportion FDPm(T((Gm)) 
actually achieved by procedure T from the convergence in distribution of the 
centered empirical processes associated with Gq.™ a-nd Gi.m, which is a conse- 
quence of Donskcr's theorem [27]: 

Theorem 3.1 (Donskcr). If the p-values are independent, then 




on [0, 1], where Zq and Zi are in- 



dependent Gaussian processes such that Zq == B and Zi =■* Bo Gi, and B 
is a standard Brownian bridge on [0, 1] . 
(ii) i/m(Gm — G) ^ Zi on [0, 1], w/iere Z = ttoZq -|- (1 — 7ro)Zi is a Gaussian 
process with continuous sample paths and covariance function given by 

E[Z(s)Z(t)] = 7r27o(s,t) + (l-^o)'7o(Gi(s),Gi(0), 

where 70 is the covariance function o/B, that is, 70 : (s, <) 1— > s A i — st. 

Theorem 3.2 (Asymptotic distribution of FDP„i for procedure T). Let T be 
a threshold function, t* ~ T{G), and p{t) ~ the positive False Discovery 
Rate at threshold t. Under Condition C.l, 

(i) 

V^(r(G„o-r*)) ^rG(z), 

(ii) 

lim FDR™(T(G,„)) = p(r*), 

m— *oo 

(Hi) 



(FDP,„(r(G„0) - p(t*)) - X , 
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with 

x.p,..)(i-P,0)(?=^-|^)+P(.-)t.,z, 

and Z = ttoZq + (1 — 7ro)Zi, where Zq and Zi are independent Gaus- 
sian processes such that Zq == B and 7L\ ="* B o Gi, and B is a standard 
Brownian bridge on [0, 1] . 

According to (zi), the asymptotic FDR achieved by procedure T is the pFDR 
at the asymptotic threshold t* = T(G). This is true because r* is positive (by 
Condition C.l). In particular, Theorem 3.2 provides a necessary and sufficient 
condition under which a multiple testing procedure with Hadamard differen- 
tiable threshold hmction asymptotically controls FDR: 

Corollary 3.3. A threshold function T satisfying Condition C.l asymptotically 
controls FDR if and only if its pFDR at t* = T{G) (i.e. its asymptotic FDRj is 
below OL, that is, if and only if 

< a . 



Remark 3.4 (Form of 7^). The expression of 7^ for threshold functions is given 
by Corollary 7.12, which shows that for one-stage adaptive procedures (where 
the level function A is constant), Tq is proportional to the inverse of the differ- 
ence between the slopes of and G at r*. For two-stage plug-in procedures, 
which typically estimate tto using G{uo) for some uq (e.g. uq = X for procedure 
Sto02), Tg involves an additional term that depends on G(mo), and the asymp- 
totic distribution of the FDP depends on the centered Gaussian random variable 
Z(mo), where Z is defined in Theorem 3.1. 



3.2. Asymptotically equivalent procedures 

Some multiple testing procedures cannot be written in terms of threshold func- 
tions, because they do not depend exclusively on Gm, but instead also directly 
depend on the number m of observations. When such procedures are only slight 
perturbations of actual threshold procedures, they share the same asymptotic 
distribution, as explained below. 

Definition 3.5 (Asymptotic equivalence of multiple testing procedures). Let T 

be a threshold function for which Condition C.l holds for T. A multiple testing 
procedure M is asymptotically equivalent to T as m ^ -too if and only if 

^/^(FDP„ {M{Pu ■ --Pm)) - FDR„. (r(G„))) ^0. 

Proposition 3.6 (Asymptotic equivalence of thresholding procedures). Let T 

be a threshold function, and e = {em)meN a positive sequence. For m £ N, let 
%n: D[0,1]^ [0, 1] such that 

yP eD[0,l],T{F- £,„) < 7;„(F) < T{F) . 
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// Condition C.l holds for T, and if Sm = o %n is asymptotically equiv- 

alent to T as m ^ +00. 

Several applications of Proposition 3.6 are given in section 4. For example, the 
asymptotic behavior of procedure 7m = STS04(A) can be derived from that of 
procedure T = Sto02(A), for which Theorem 3.2 may be used because Sto02(A) 
is an actual threshold function. 



3.3. Regularity conditions 

For the threshold functions under investigation. T(G) is defined as the last 
point for which G > r(A{G), ■). Therefore, the existence of a unique interior 
right crossing point between G and r{A{G), •) ensures that Theorem 3.2 and 
Proposition 3.6 are applicable, i.e. that T(G) > 0, and that T is Hadamard 
differentiable at G (Condition C.l). For two-stage adaptive (plug-in) procedures, 
for which the level function A is not constant, additional technical assumptions 
concerning the regularity of A require checking (see Corollary 7.12) to ensure 
that Condition C.l holds. 

Definition 3.7 (Right crossing point). Let Va be a rejection curve, and A a 
level function. Denote hyT : F i-^hi{F,A{F)) the associated threshold function, 
where U{F,a) = sup{it G [0, l],F{u) > ra{u)}. A right crossing point for the 
multiple comparison problem defined by T ( or, in short, a right crossing point 
for T), is a point t e [0, 1] such that G{t) = r„(t), and g(t) < |^ iA{G), t). If 
t belongs to the open interval (0, 1) it is called an interior right crossing point 
for T. 

Condition g{t) < ^ {A{G),t) in Definition 3.7 ensures that G and ''^(g) = 
r{A{G),-) actually cross at t, i.e. that G > M(G) in ^ left-neighborhood of i, 
and that G < 7-_4(g) in a right-neighborhood of t. 

Studies of the asymptotic distribution of the abovcmcntioncd FDR controlling 
procedures require investigation, in each case, of the conditions guaranteeing the 
existence of a unique interior right crossing point. To this end, we broke this 
condition down as follows: 

Condition C.2 (Existence). T has an interior right crossing point. 

Condition C.3 (Uniqueness). T has at most one interior right crossing point. 

Condition C.3 always holds for procedures based on Simes' line (BH95, Sto02, 
and BKY06) because their rejection curve is linear, and G is concave. Condi- 
tion C.2 typically holds in situations in which the slope of G at the origin is large 
enough. In the case of the BH95 procedure, Chi recently showed the existence 
of a critical value a* depending solely on the distribution function G of the 
p-values, such that if a < a* , the number of discoveries made by the BH95 pro- 
cedure is stochastically bounded as the number of tested hypotheses increases, 
whereas if a > a*, the proportion of discoveries converges in probability to a 
positive value r* = T(G) [-5]. 
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In section 4, we provide a detailed analysis of a number of FDR eontroUing 
procedures, and present, for each, a critical value for the target FDR level char- 
acterising situations in which condition C.2 is guaranteed for the procedure. 

4. Results for procedures of interest 

We apply the results of the preceding section to a series of procedures with 
proven (asymptotic) FDR control. Starting from the original BH95 procedure 
and its Oracle version (section 4.1), we then turn to adaptive procedures, which 
implicitly or explicitly incorporate an estimate of the proportion ttq of true 
null hypotheses: one-stage adaptive procedures are studied in section 4.2, and 
two-stage adaptive procedures (also called plug-in procedures) are studied in 
section 4.3. 

4.1- BH95 procedure 

We will first recall the definition of the BH95 procedure in our framework. 

Definition 4.1 (Procedure BH95[1]). The BH95 procedure is the multiple test- 
ing procedure with threshold function 



As the rejection curve of procedure BH95 is linear, and G is concave, the 
uniqueness Condition C.3 always holds, and the existence Condition C.2 can be 
reduced to a > a*, where a* = inf„^o 'u/G(it) = limu^o VsC'^) corresponds to 
the critical value of the BH95 procedure ['>]: 

Condition C.4 (Condition C.2 for the BH95 procedure). The target FDR level 
a is greater than the critical value a* of the BH95 procedure. 

The criticality phenomenon is illustrated in Figure 2 for Laplace (double 
exponential) test statistics. The Weak Law of Large Numbers phenomenon an- 
alyzed by [1], which occurs when a > a*, was noted by [10]. We now derive 
the corresponding central limit theorem under the same hypothesis, and the 
asymptotic distribution of the FDP actually achieved by the BH95 procedure. 

Theorem 4.2 (Asymptotic properties of the BH 95 procedure). Letr* =T^'^^^(G). 
Under Condition C.4, 



with Z = ttoZq + (1 — 7ro)Zi, where Zq and TL\ are independent Gaus- 
sian processes such that Zq == B and Zi ''^ B o Gi, and B is a standard 
Brownian bridge on [0, 1]. 



(F) = sup{M e [0, l],F{u) > u/a} . 
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Figure 2. Critical value of the BH95 procedure for Laplace test statistics with location pa- 
rameter 6 = 2, and ttq = 0.5. Solid line: distribution function G; straight lines: Simos' lines for 
several values of a. There is an interior right crossing point between the distribution function 
of the p values and Simes' line if and only if a > «* = l/g(0). 



(ii) 

^ (FDP„(rB"95(g^^)) _ ^^^^ ^ _^ (^^ 

Applying the BH95 procedure at level a/nQ leads to an Oracle procedure (as 
ttq is not known) that is more powerful as it controls FDR at level exactly a. 
This procedure, which we denote by BH95o, has threshold function 

^BH95o(^) = SUp{M e [0, 1], > TTow/a} , 

and its critical value is therefore iroa* , which translates into the following reg- 
ularity condition: 

Condition C.5 (Condition C.2 for the BH95 Oracle procedure). The target 
FDR level a is greater than t^qO* , where a* is the critical value of the BH95 
procedure. 

The corresponding asymptotic properties can be derived from Theorem 4.2: 

Corollary 4.3 (Asymptotic properties of the BH95 Oracle procedure). Let 

T* ^r^^'^'^°{G). Under Condition C.5, 



-BH95o/ 
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(n) 



with Z = ttoZq + (1 — 7ro)Zi, where Zq and Zi are independent Gaus- 
sian processes such that '. 
Brownian bridge on [0, 1] 



sian processes such that Zq == B and Zi ^= B o Gi , and B is a standard 



4-2. One-stage adaptive procedures 

The first class of adaptive procedures studied here are one-stage adaptive pro- 
cedures, because they estimate ttq imphcitly, rather than through a level func- 
tion A. 

Definition 4.4 (Adaptive procedure). Let : [0, 1] [0, 1]. The adaptive pro- 
cedure associated with r^ is the multiple testing procedure defined by the threshold 
function 

r(F) =sup{ue [0,l],F(u) >r„(u)} . 

The rejection curve of adaptive procedures is not linear, so the conditions un- 
der which Condition C.l is fulfilled are more subtle than for the BH95 procedure 
(section 4.1) or for two-stage adaptive procedures (section 4.3). 

Procedure FDR08(A). The rejection curve of the FDR08 procedure [7] is de- 
fined for u e [0,1] by fa{u) ^ a+{'^-c)u ■ •^"'^^^ ^ ^^'^ corresponding 
threshold function is always equal to 1. This procedure therefore systematically 
rejects all hypotheses, and does not control FDR either for finite sample size or 
asymptotically. Several ways of overcoming this problem have been proposed [7] , 
including truncating the rejection curve, yielding the following procedure: 

Definition 4.5 (Procedure FDR08(A)). Let X G [0, 1). The rejection curve of the 
FDR08(A) procedure is defined by f^{u) ~ fa{u) for u < X, and -\-oo otherwise. 
The threshold function of the FDR08(A) procedure is therefore given by 



^hDK08(^) = sup iu e [0, X],F{u) > 



a + (1 — a)u J 
We introduce the following regularity condition: 
Condition C.6. A > k, where k = , 

Note that ^k, \~ a" ^ the crossing point between the rejection curve fa and 
the distribution function DU(7ro) in the extremal Dirac-Uniform configuration 
where all p-values drawn from Hi are equal to 0. As G = ttoGq + (1 — 7ro)Gi < 
DU(7ro), condition C.6 ensures that any interior right crossing point between G 
and fa occurs before A. In practice, k is unknown because it depends on ttq. 
However, an upper bound for k can be deduced from a lower bound for ttq; for 
example, in microarray data analysis, it can often be assumed that ttq > ^: in 
this case, k is smaller than -r^- 
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By definition 4.5, the rejection curve of any procedure FDR08(A) satisfying 
Condition C.6 is equal to fa on [0, k], corresponding to the admissible region 
for interior right crossing points. The following Proposition is a straightforward 
consequence of this observation: 

Proposition 4.6. All FDR08(A) procedures satisfying Condition C.6 are asymp- 
totically equivalent in the sense of Definition 3.5. 

As the corresponding asymptotic distribution does not depend on A, we will 
refer to it simply as the "asymptotic distribution of the FDR08 procedure". In 
order to characterize this distribution we introduce a further technical condi- 
tion to ensure that k < 1. Combined with Condition C.4, it also ensures that 
existence Condition C.2 holds for procedure FDR08(A), because the slope of 
at the origin is \/a. 

Condition C.7. a <ttq. 

Condition C.7 is a mild assumption in practice, because ttq is typically ex- 
pected to be greater than 1/2, in microarray data analysis, for example. When 
oi > ttq, there is no need for sophisticated FDR controlling procedures because 
rejecting all hypotheses yields FDP = ttq and thus FDR < a. 

Theorem 4.7 (Asymptotic behavior of procedure FDR08). Let A G [0, 1) such 
that Condition C.6 is fulfilled, and r* = sup |w G [0, k], G{u) > a+{i-a)u } • 
der uniqueness Condition C.3, and existence Conditions C.4 and C.7, we have 

with ttq (r*) = -Uj^^i^-^, and 

T* Gl[T*) 

where p* = aTTo/Tfo (t*) is the pFDR achieved by procedure FDR08, 

(l-7f^(r*))7f^(T*)/a 



7ro(T*)2/a - g{T*) 



Zq and Zi are independent Gaussian processes such that Zo == B and TL\ =' 
B o Gi, and B ?s a standard Brownian bridge on [0, 1]. 

As 7fo(T*) — i^^^]^^*^ G [7ro,l], we have Ti^a < p* < a, so that procedure 
FDR08 is asymptotically more powerful than procedure BH95, and less powerful 
than procedure BH95o. 



Procedure BR08(A). 
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Definition 4.8 (Procedure BR08(A) [4]). Let A e [0,1). The rejection curve 
of the BR08(A) procedure is defined by b^{u) = a{i-x)+u 1^ o.'f^d +oo 

otherwise. The threshold function of the BR08(A) procedure is therefore given by 



u e [0,A],F(u) > 



a{l — A) + u 



Procedure BR08(A) is actually defined by the rejection curve (l + ^) 6^ [4]. 
However these procedures are asymptotically equivalent according to Proposi- 
tion 3.6; we will therefore use Definition 4.8. 

As for the FDR08 procedure, the rejection curve of the BR08(A) procedure 
is not linear and we therefore need to make two assumptions to ensure that 
existence Condition C.2 holds: Condition C.8 ensures that there is no criticality 
phenomenon, that is, that the slope of the distribution function G is great 
enough at the origin, and Condition C.9 ensures that a right crossing point 
occurs before A, because the BR08(A) procedure is truncated at A: 

Condition C.8. The target FDR level a satisfies a(l — A) > a* , where a* is 
the critical value of the BH95 procedure. 

Condition C.9. The distribution function G satisfies 

A 1 - G(A) 



G(A)< 



1-A 



Remark 4.9. Condition C.9 may be written as G(A) < 6^, or as G(A) < /„, 
because the rejection curves of procedures BR08(A) and FDR08 intersect at A. 

Theorem 4.10 (Asymptotic distribution of procedure BR08(A)). Let A G [0, 1) 
and T* = T^'^''^^^^ (G) . Under uniqueness Conditions C.3 and existence Condi- 
tions C.8 and C.9, we have 

(fDP„(T^R°«W(G™)) - aTTo^^^) - X^-^^s^^), 

with 

^BR08(A) ^ p.(i _ p*C(r*))?^^ - p'^(l - p*)C(r'^)^, 

r* Gi(t*) 

where p* = qtto j^Jg^^*) is the pFDR achieved by procedure BR08(A), 

G(r*)Vr* 



Gir*){l-G{r*))/r* -sir*) ' 



Zq and Zi are independent Gaussian processes such that Zo == B and Zi =^ 
B o Gi, and M is a standard Brownian bridge on [0, 1]. 

Theorem 4.10 implies that procedure BR08(A) controls FDR asymptotically 
at level a: as r* < A, we have p* < otto j-i^p^, which is smaller than a because 

'^{^) ~ ^~i^^x^ upper bound for ttq. 
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However, as ^^(ii) > u/a if and only if it > Aa, procedure BR08(A) need not 
be more powerful than procedure BH95, and we have the following characteri- 
zation: 



where 3> means "is more powerful than" , and Tg^gg is the asymptotic threshold 
of procedure BH95. An explicit characterization of situations in which BROS (A) 3> 
BH95 for Gaussian test statistics is given in [4]. 

4.3. Two-stage adaptive (plug-in) procedures 

In this section we study two-stage adaptive or plug-in procedures, in which a 
conservative step-up procedure is applied to a data-dependent level. In par- 
ticular, we consider the case of Simes' line-based plug-in procedures, in which 
procedure BH95 is applied at level a/yro, where ttq is estimated from the data: 

Definition 4.11 (Simes' line-based plug-in procedure). Let A : D[Q, 1] R!j_. 
The Simes' line-based plug-in procedure associated with A is the multiple testing 
procedure defined by the threshold function 



Such procedures will simply be called plug-in procedures hereafter. 

As Tq, is linear, and G is concave, uniqueness Condition C.3 always holds for 
plug-in procedures, and existence Condition C.2 is the same as for procedure 
BH95, except that a is replaced by the value of the level function A at G: 

Condition C.IO (Condition C.2 for plug-in procedures). The level function 
A{G) associated with the target FDR level a is greater than the critical value of 
the BH95 procedure. 

Care is required when deriving the asymptotic distribution of the FDP for 
plug-in procedures, because the Hadamard derivative of T, 7'g{H), typically 
involves the value of H at r* and at a point m(A) used for the estimation of ttq: 
u(A) = A for procedure Sto02, and u{X) = U (G, A) for procedure BKY06(A). 
The asymptotic variance of the False Discovery Proportion therefore involves 
the covariance between Z(t*) and Z(w(A)). 

Theorem 4.12 (Asymptotic FDP for procedures based on Simes' line). Let 
T : F 1-^ U{F,A{F)) a threshold function based on Simes' line. If the level 
function A is Hadamard-difjerentiable at G, tangentially to C[0, 1], and satisfies 
existence Condition C.IO, then 



withJj = 7roZo + (l — 7ro)Zi, whereZo andZi are independent Gaussian processes 
such that Zq B and Zi B o Gi, and M is a standard Brownian bridge on 



BR08(A) > BH95 



TBH95 > O-X , 





[0,1]. 
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Wc consider the two types of plug-in procedures most widely used and the- 
oretically justified: Sto02-like procedures (Sto02 [21], STS04 [23]), in which ttq 

is estimated by or a slight variant, and the BKY06 procedure [2]. in 

which an upper bound for ttq is derived from a first application of the classical 
BH95 procedure. 

Procedure Sto02. 

Definition 4.13 (Procedure Sto02 [21]). Procedure Sto02 is the multiple testing 
procedure with threshold function 

u 1 - F{\) 



TSt°02W(F) = sup (t. e [0, l],F{u) >- — 

a 1 



X 



The level function of this procedure is therefore 

a 



AiF) 



with 

7fo'^(A) will simply be denoted byno{\). 

Condition C.ll (Condition C.2 for procedure Sto02(A)). The target FDR level 
a is greater than ttq (A) a* , where a* is the critical value of the BH95 procedure. 

This procedure is known to provide asymptotic control of FDR at level a [21], 
but does not necessarily control FDR at level a for finite sample size. This led 
to the definition of a modification of the Sto02 procedure that does control FDR 
even for finite sample size [2-)]: 

Definition 4.14 (Procedure STS04(A) [2;!]). Procedure STS04(A) rejects p- 
values smaller than 

r^™^(''(G„0 = supL e [QAl^M > ^i±4^^^| . 

I a 1 — A I 

According to Proposition 3.6, procedures STS04(A) and Sto02(A) are asymp- 
totically equivalent provided that Conditions C.9 and C.ll hold (see Proposi- 
tion 7.16 page 1104 for a formal proof). 

Theorem 4.15 (Asymptotic properties of the Sto02/STS04 procedure). Let 

A € (0, 1), and t* = T^*°°^'-^) (G). Under existence Condition C.ll, we have 

f FDP„(rSt°02(G„)) - - 

V 7^0 (A) J 

where 

,,c;t„n9 TTno; /Zo(r*) , Z(A) 
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withZ, = 7roZo + (l — 7ro)Zi , where andZi are independent Gaussian processes 

such that Zq == B and Zi =■* B o Gi, and M is a standard Brownian bridge on 
[0, 1] . X'^'^°^^ is therefore a centered Gaussian random variable, with variance 

noayn-T* VarZ(A) r* A A - r*A 1 

l^^+(l-G(A))2+ ''V*(1-G(A))/' 

where 

VarZ(A) = ^gA(l - A) + (1 - ttq)' Gi(A)(1 - Gi(A)) . 
Corollary 4.16. If t* < X, 



W^{X)J { T* (1-G(A))2 7f^(A) 

As ttq (A) = TTo + (1 — Ti'o)— we have, for any A < A', ttq < ttq (A') < 
7f^(A) < 1, BH950 > Sto02(A') > Sto02(A) > BH95. 

Procedure BKY06. Letting (3 ~ procedure BKY06 involves applying 

procedure BH95 at level i^ji^py^n ' where R{(3) is the number of hypotheses 
rejected by a first application of the BH95 procedure at level /?. We shall consider 
a recently proposed generalization of this procedure [4], in which procedure 
BH95 is applied at level i^]i(xy,n Ot. The original BKY06 procedure corresponds 
to A = 

1+a 

Definition 4.17 (Procedure BKY06(A)[j]). Let A e [0, 1), and 

^^^^ = "l-F(Z^(F,A)) ' 

where 

ZY(i^,A) =sup{«e [0,l],i^(w)> ^} . 

The threshold function of procedure BKY06(A)js defined for any F € £'[0, 1] 
byT^'^^^^'^^\F)=U{F,A{,F)), that is, 

^BKY06(A)(^) = sup e [0, l],F{u) > ^ ^~P^^^P^>^)) 
[ a 1 — A 

Remark 4.18. As the proportion R{X)/m of hypotheses rejected by procedure 
BH95 at level A equals Gm(Z^(G„i, A)) (see Proposition 7.8 page 1098), A{Gm) 
may be written as a i_]{(^x)/m • 

Remark 4.19. The exact definition of procedure BKY06(A) adds 1/m to the 
denominator of the level function: 

"^^^^ ^ "l + ;^-^(W,A)) ' 

which permits proving that this procedure controls FDR for finite sample size [4]. 
According to Proposition 3.6, these two procedures are asymptotically equiva- 
lent, so we will use Definition 4.17. 
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As procedure BKY06(A) is based on two successive applications of procedure 
BH95, at level A and a(l — A), Condition C.2 holds if and only if Condition C.8 
holds and A > a"^. 

Condition C.12. The parameter A satisfies \ > a* , where a* is the critical 
value of the BH95 procedure. 

Theorem 4.20 (Asymptotic properties of the BKY06(A) procedure). Let a G 

[0,1], and A G [0,1). Let u{X) ~ U{G,X) be the asymptotic threshold of the 
BH95 procedure applied at level X, and t* = T^^^*^^*-"^' (G). Under existence 
Conditions C.8 and C.12, 

^ f FDR^(T^^^°e(^)(G„)) - - X^^^°'i>^\ 

where 

T^BKY06(A) ^ 7roQ;(l - A) / Zo(t*) 1 Z(m(A)) a 

1 - G{u{X)) \ r* ^ 1 - a(l - X)g{u{X)) 1 - G{u{X)) ) ' 

withlj ~ 7roZo + (l — 7ro)Zi , where and Zi are independent Gaussian processes 

such that Zq B and Zi B o Gi, and M is a standard Brownian bridge on 
[0,1]. 

As m(A) is the asymptotic threshold of the BH95 procedure applied at level A, 
we have G(m(A)) = m(A)/A, u{X) < X. Therefore, i_q'(^^(\)) — i-g(A) ' ^^"^ ^^'^ 
asymptotic level of procedure BKY06(A) is less than a because ^"I'Zx'^ ^ ^o- 

However, as for procedure BR08(A), procedure BKY06(A) need not be more 
powerful than BH95: a comparison of the asymptotic FDR for these two pro- 
cedures shows that situations in which BKY06(A) S> BH95 are characterized 
by G{ux) > A, that is, G^(ua) > u\ because G{ux) = ux/X. Hence, BH95 S> 
BKY06(A) corresponds to situations in which G is too close to the Uniform dis- 
tribution. For example, if G{x) < y/x for all x € [0, 1], then for any A G [0, 1], 
BH95 > BKY06(A). 

5. Connection between one-stage and two-stage adaptive procedures 

We have introduced two types of FDR controlling procedures generalizing the 
BH95 procedure: two-stage adaptive (plug-in) procedures explicitly incorporate 
an estimate of ttq into the standard BH95 procedure, whereas one-stage adaptive 
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procedures do not explicitly use such an estimate, but still provide tighter FDR 
control than the BH95 procedure. 

We will now investigate connections between one-stage and two-stage adap- 
tive procedures, which naturally appear when using the formalism of threshold 
functions: with a striking symmetry, the threshold of procedure BR08(A) may 
be interpreted as a fixed point of an iterated BKY06(A) procedure, whereas 
the threshold of procedure FDR08 may be interpreted as a fixed point of an 
iterated Sto02(A) procedure. We provide heuristic reasons for these connections 
in section 5.1; in section 5.2 we present general results for the connection be- 
tween one-stage and two-stage adaptive procedures, and derive consequences for 
the connection between procedures Sto02(A) and FDR08 on the one hand, and 
between procedures BKY06(A) and BR08(A) on the other hand. 



5. 1 . Heuristics 

Procedures BKY06(A) and BR08(A). The BKY06 procedure was designed to 
derive an approximate upper bound for ttq from a first application of procedure 
BH95, and to use this upper bound in a second application of the BH95 pro- 
cedure, leading to less conservative FDR control. For A S [0,1), the threshold 
function of the BKY06(A) procedure is defined by 

^BKY06(A)(^) = sup(. e [0,1], > ll^lMEim , 

1^ a 1 — A J 

where Z^(F, A) = sup {u e [0, 1], > f }• It therefore seems natural to iterate 
this process, using the number of rejections at the second application to find a 
less conservative upper bound for ttq, and to use this new upper bound in a third 
application of the BH95 procedure, and so on. Based on this idea, Benjamini 
et al. suggested defining a multi-stage procedure for the particular situation 
in which A ~ [2]. In our framework, this iterative process suggests the 

introduction of a fixed-point procedure defined for any F S -D[0, 1] by: 



r^^''°'(^Hi^) = sup|ziG [0,l],F(u)> 



u 1 - F{u) 



a 1 — A 

The term fixed-point procedure refers to the following property of the corre- 
sponding asymptotic threshold = T^^'^^'^^\g) . Let us suppose that is 
the threshold obtained at a given stage of the abovementioned iteration pro- 
cess. As G(t^) = t^(1 — G{t^)) / a{l — A), is also the asymptotic threshold 
at the next stage, and is thus a fixed point of the iteration process. It turns 
out that this fixed-point procedure is the BR08(A) procedure investigated in sec- 
tion 4.2: F{u) > ^j(Yrxy(l — F{u)) may be written as F{u) > -^y^^T^^ ^^'^ 
right-hand side is the rejection curve 5^ of the BR08(A) procedure. 

Procedures Sto02(A) and FDR08(A). The same idea may be adapted to 
procedure Sto02(A), which is defined for < A < 1 by the threshold function 

ul-F{X)' 



rSt°°2(^)(F) = sup |w e [0, l],F{u) > 



a 1 - A 
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If T\ = T^'°°^^^'' (Gm) denotes the empirical threshold of procedure Sto02(A), 
one may use tx to estimate ttq, that is, calculate the threshold given by procedure 
Sto02(TA), and so on. This suggests that an associated fixed-point procedure 
could be defined as 

r^-°2(F) = sup L e [0, 1], ^^(.) > !ii_iZM j . 

Again, the term fixed-point procedure refers to the fact that if — T{G) 
is used as a new A to estimate ttq in procedure Sto02(A), then the asymp- 
totic threshold of procedure Sto02(A) is also r^, which is therefore a fixed 
point of the iteration process. It turns out that this fixed-point procedure is the 
FDR08 procedure investigated in section 4.2: F{u) > ^ ^^^i"^ ^^-y be written 
as F{u) > , and the right-hand side is the rejection curve fa of the 

FDR08 procedure. 



5.2. Formal connections 



We present a general result concerning connections between one-stage and two- 
stage adaptive procedures, providing a formal justification for the connections 
mentioned in section 5.1, and accounting for their symmetry. This result is based 
on the following assumption concerning the threshold function of the one-stage 
adaptive procedure: 

Condition C.13. There is a curve Ca ■ D[0,1\ x [0,1] such that the threshold 
function T may he written as 

T{F) =sup{itG [0,A],F(it) >Cc{F,u)} , 

where u i-^ Ca{G, u)/u is non increasing on [0, A]. 

Remark 5.1. In Condition C.13, Ca{F, •) is not the rejection curve of procedure 
T, because it depends on F. For example, for procedure FDR08, we will use 

u 1 - F{u) 



,{F,u) 



a 1 



Theorem 5.2 shows that we can associate with a one-stage adaptive procedure 
fulfilling Condition C.13 a two-stage adaptive procedure with linear rejection 
curve, and level function given by 

for fixed t G (0,1). The asymptotic threshold of the one-stage procedure may 
then be interpreted as the fixed point of iterations of the two-stage procedure. 

Theorem 5.2 (Connection between one-stage and two-stage adaptive proce- 
dures). Let A G (0, 1). Let us consider a multiple testing procedure with a thresh- 
old function T that may be written as 



T{F) = sup {u e [0, A], F(m) > Cc(F, u)} 
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for any F € I?[0, 1]. Let Tt he the threshold Junction defined by 
Tt{F) = sup L e [0, l],F{u) > EdElIlu 



for any t G (0,1) and any F G _D[0, 1]. Let us assume that existence Condi- 
tion C.2 and uniqueness Condition C.3 hold for procedure T , and that, for any 
t G (0, 1), existence Condition C.2 holds for procedure 7^. Let t* = T{G) and 
T{t) = TfiG) be the asymptotic thresholds of procedures T and Tf, respectively. 
If Ca satisfies Condition CIS, we have 

(i) for any t G (0, A], 

ft < r* =^ T{t) G [t,r*] 
|t > r* ^ r(t) G [r*,i] 

(ii) Let t G (0, A]. Define the sequence (tn) G [0, 1]^ by to = t, and t^+i = T{ti) 
forieN. Then 

lim tn — T* . 

n — >oc 

Corollary 5.3 (Asymptotic power comparison). With the same notation and 
under the same conditions, the following assertions are equivalent: 

(i) Procedure % is asymptotically more powerful than procedure T 
(li) T{t) > 
(Hi) t> T* 
(iv) t > r(t) 

In the remainder of this section, we use Theorem 5.2 to characterize the 
connection between the abovementioned procedures. 

Procedures Sto02(A) and FDR08(A). Theorem 5.4 gives the convergence of 
the process consisting of the recursive use of the asymptotic threshold of pro- 
cedure Sto02(A) as a new A. It holds under the same regularity conditions as 
those required to obtain the asymptotic distribution of procedure FDR08. 



Theorem 5.4 (Connection between procedures Sto02(A) and FDR08). Let k 

r* = sup i u G [0, k\,G{u) > 



, and 

u 1 - G{u 



a 1 — u 

he the asymptotic threshold of the FDROB procedure. For u G [0, 1], let 



t{u) = sup < u G [0, 1], G(it) > 



u 1 - G{\) \ 
a 1 - A J 

he the asymptotic threshold of procedure Sto02(u). For any t G (0, 1), define the 
sequence {tn) G [0, 1]^ by to = t, and t^+i = T{ti) for i E N. Let us assume that 
uniqueness Condition C.3 holds for procedure FDROB, and that the target FDR 
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level a satisfies existence Conditions C.4 and C.7. Then, 

lim tn — T* . 

n — ^oo 

Corollary 5.5 (Asymptotic power comparison — Sto02(A) vs FDR08). With 
the same notation and under the same conditions, procedure Sto02(A) is asymp- 
totically more powerful than procedure FDR08 if and only if X > t{X). 

When using procedure Sto02(A) in practice, we would not want any of the 
rejected hypotheses to be incorporated into the estimation of ttq. Thus, the 
empirical rejection threshold T^*°°^^'^' ((G,„) should be less than A. In such sit- 
uations, as rS'°°2(^)(G„) converges at rate l/^/m to r(A) = rS'°°2(^) (G)^ pro- 
cedure Sto02(A) is probably more powerful than procedure FDR08 according to 
Corollary 5.5. 

Procedures BKY06(A) and BR08(A). Theorem 5.6 characterizes the connec- 
tion between procedure BKY06(A) and procedure BR08(A). It holds under the 
same regularity conditions as those required to obtain the asymptotic distribu- 
tion of procedure BR08(A). 

Let T* = T^^°^W{G) be the asymptotic threshold of the BR08(A) procedure. 
Under uniqueness Condition C.3 and existence Conditions C.8, C.9 and C.12, 
(tn) is non decreasing, and converges to t* . 

Theorem 5.6 (Connection between procedures BKY06(A) and BR08(A)). Let 

A e (0, 1). For F e ^[0, 1] and [i G [0, 1], let 

Z^(F,/3)=sup|ue [0,l],F(u)> ^1 . 

For any u G [0,1), let t{u) = lA (^G, "Ig^^^ ^ . With this notation, T{G) = 

t(w(A)) is the asymptotic threshold of the BKY06(A) procedure, where u[X) = 
U{G,\). Let 

r* =sup(iie [0,A],G(ii) > \ ] 

[ a(l - Aj + u J 

be the asymptotic threshold of the BR08(A) procedure. Define the sequence (tn) G 
[0, 1]^ by to = u{X), and ti+i = T{ti) for i G N. Let us assume that uniqueness 
Condition C.3 holds for procedure BR08(A), and that the target FDR level a 
satisfies existence Conditions C.8 and C.9. Then 

lim t„ = T* . 

n — >oc 

Corollary 5.7 (Asymptotic power comparison — BKY06(A) vs BR08(A)). With 
the same notation and under the same conditions, procedure BR08(A) is asymp- 
totically more powerful than procedure BKY06(A) if and only if the asymptotic 
threshold t* of procedure BR08(A) satisfies r* > A — a(l — A). 

For example, setting A to a value less than corresponding to the orig- 
inal BKY06 procedure [2], ensures that the associated BR08(A) procedure is 
asymptotically more powerful than the associated BKY06(A) procedure. 
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6. Concluding remarks 

This paper demonstrates the power and flexibihty of the formaUsm of threshold 
functions, making it possible to derive the asymptotic properties of well known 
FDR controlling procedures with their associated regularity conditions, and to 
identify and characterize novel connections between one-stage and two-stage 
adaptive procedures. These results are summarized in Table 1. We should recall 
that the threshold function associated with the level function A and rejection 
curve Ta = r[a, •) is defined by 

r(F) = sup{uG [{),l],F{u)>r{A{F),u)} . 

By definition, the level function A equals a for one-stage procedures, and the 
rejection curve of Simes' line-based procedures is : u i-^ u/a. 

Regularity conditions. For one-stage adaptive procedures FDR08 and 
BR08(A), the uniqueness Condition C.3 has to be assumed (cf. Table 1): as 
the rejection curve is not linear, the interior right crossing point is not neces- 
sarily unique; in practice the uniqueness condition holds except in pathological 
situations. For Simes' line-based procedures BH95, Sto02(A) and BKY06(A), ex- 
istence Condition C.2 holds provided that the slope of the distribution function 
exceeds a certain threshold at the origin (that is, that there is no criticality 
phenomenon). For one-stage adaptive procedures, it is also required that the 
rejection curve ends below the distribution function G, which corresponds to 
Condition C.7 for procedure FDR08, and Condition C.9 for procedure BR08(A). 

The criticality phenomenon studied by [5] is intrinsic to the multiple testing 
problem, and not specific to a given procedure, as the minimum attainable pFDR 
level (3* = inft>o pFDR(t) depends solely on the parameters of the model [5]. 
When = say for the Gaussian location problem, there is no criticality 
phenomenon for any procedure: a* = Q, and all existence Conditions concern- 
ing the behavior of the distribution function G close to are fulfilled for any 
procedure, and for any target FDR level a. When (3* > 0, say for the Laplace 



Table 1 

Comparison of FDR controlling procedures, characterized by their level function A. and their 
rejection curve ra ■ Conditions for the existence and uniqueness of an interior right crossing 
point are recalled, together with the corresponding pFDR relative to that of the BH95 

procedure: ttoo 



Name 


BH95 [ ] 


FDR08 [ -] 


BR08(A) [ ] 


Sto02(A) [ ] 


BKY06(A) [ ] 


A{F)/a 


1 


1 


1 


l-A 
l-F(A) 


l-A (a) 
1-G„(«,) 




u/a 


(6) 

OL-\-{l — a)u 


u (c) 
a(\-X) + u 


u/a 


u/a 


Existence 


C.4 


C.4 & C.7 (''^ 


C.8 & C.9 f"*' 


C.ll 


C.12 


Uniqueness 




C.3 


C.3 






pFDR/TToQf 


1 


■'■"'^FDRoa 


l-A 


l-A 


l-A (a) 


1-G(-F*DR08) 




l-G(A) 


I-G(ua) 



(a) : ux is the asymptotic threshold of the BH95 procedure at target level A; (b) : truncated at 
"^(i-a) ' ' ti'uncated at A; {d) : Sufficient (not necessary) conditions. 
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location problem (Figure 2, page 1076), there is a criticality phenomenon for 
every procedure; however the critical value, that is, the minimum target FDR 
level for which existence Condition C.2 holds, may depend on the procedure, as 
illustrated by the existence conditions in Table 1. 

Povirer comparisons. All procedures are asymptotically conservative, and 
therefore yield asymptotic FDR below the target level. Procedures FDR08 and 
Sto02 (and thus STS04) are always more powerful than procedure BH95, but this 
is not the always the case for procedures BR08(A) (section 4.2) and BKY06(A) 
(section 4.3). 

For one-stage adaptive procedures, for any A G (0, 1) such that the regularity 
conditions for procedures FDR08 and BR08(A) hold, FDR08 is asymptotically 
more powerful than BR08(A). Indeed, Condition C.9 ensures that the asymptotic 
thresholds of both procedures are less than A. As the rejection curve fa of 
procedure FDR08 is smaller than the rejection curve 5^ of BR08 on [0,A], the 
asymptotic threshold of procedure FDR08 is greater than that of procedure 
BR08(A). However, it should be noted that procedure BR08(A) does control 
FDR for a finite number of tested hypotheses, whereas procedure FDR08 does 
not. 

For two-stage adaptive procedures, for any A € (0, 1) such that the regu- 
larity conditions for procedures Sto02(A) and BKY06(A) hold, Sto02(A) (and 
thus STS04) is asymptotically more powerful than BKY06(A), as demonstrated 
by the corresponding asymptotic FDR levels in Table 1: as ux < A, we have 
i-C(t.x) — i-G(A) ■ "^^^^ suggests that procedure STS04(A) is preferable to pro- 
cedure BKY06(A) in practice. This recommendation should be balanced against 
the choice of A and the desired robustness to dependence between null hypothe- 
ses. Based on a simulation study, procedure Sto02(Q;) was recently reported to be 
much more robust to positive dependence between null hypotheses than proce- 
dure Sto02(l/2) [4] , which is still a standard choice in practical implementations, 
such as the SAM (Significance Analysis of Microarrays) software [24]. 

Towards optimality. This comparison raises the question of whether the 
formalism of threshold functions can be used to derive procedures more powerful 
than those studied here. One possible approach consists of trying to improve the 
estimation of ttq to build a procedure closer to the Oracle BH95 procedure, as 
discussed in [11]. However, consistent estimators of ttq have slower convergence 
rates than l/-y/m, resulting in slower convergence rates than l/y/m for the 
associated FDP. This may be illustrated by the influence of A on procedure 
Sto02(A): the larger A, the smaller the bias E [7ro(A)] — ttq, and the larger the 
variance of 7fo(A). The question of how to choose A as a function of the number 
of hypotheses tested and the assumed regularity of G is discussed in another 
work [14]. 

Another possibility would be to consider procedures more general than those 
used in this paper: the BH95o procedure has been shown to give the lowest false 
non discovery rates (FNR) of the threshold procedures controlling FDR at level 
a [10]. The question of optimality in a broader family of testing procedures has 
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recently been raised [25]: Z score-based threshold procedures may outperform 
p value-based threshold procedures, as they make it possible to choose differ- 
ent significance thresholds for positive and negative significance cutoffs. This 
suggests to extend our framework to Z score-based procedures. 

Confidence intervals. An interesting practical application of this work con- 
cerns the derivation of asymptotic confidence intervals for the FDP of a given 
procedure. Our results give explicit asymptotic distributions for the attained 
FDP, but this issue is not straightforward because these distributions depend 
on unknown quantities, including the proportion ttq, the asymptotic threshold 
FDR T* , or the distribution function G and its associated density g. These quan- 
tities should, in turn, be estimated. Bootstrapping techniques could be used for 
this purpose; we leave this question for further research. 

Extension to other dependence settings. We have derived the asymptotic 
properties of several multiple testing procedures and the associated regularity 
conditions in the situation in which p-values are independent. However, our 
formalism makes it possible to deal with any dependence situation for which 
the vector (Go,m, CJi^m) of empirical distribution functions of the p- values under 
the null and alternative hypotheses satisfies Donsker's invariance principle. For 
example, the form of the asymptotic distributions of the threshold T(Gm) and 
the associated FDP would remain the same in the conditional dependence model 
recently proposed by Wu [28]. 

7. Proof of main results 

7.1. Asymptotic FDP; general threshold functions 

In this section, we provide proofs for the results of section 3. 

7.1.1. Proof of Theorem 3.2 

The following lemma will be used in several subsequent proofs. 

Lemma 7.1. Let H E C[0,1], and Ht he a family of functions o/D[0, 1] that 
converges to H on (_D[0, 1], ||.||oo) as t 0. For any sequence (ut)t>o of [0,1] 
that converges to u E [0, 1] as t ^ 0, we have 

\im Htiut) = H{u) 
t-»o 

limi74(V) = H{u), 

where /(xq ) denotes lim^^^o_2,<a;o f{x). 
Proof of Lemma 7.1. We have 



\Ht{ut) - H{u)\ < \Ht{ut)- H{ut)\ + \H{ut)- Hiu)\ 
< \\Ht-H\\^ + \H{ut)~H{u)\ 
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\Ht{ui) - H{u)\ < \Ht{ui)-H{ui)\ + \H{ui)-H{u)\ 
< \\Ht-H\\^ + \H{ut)-H{u)\ 

as H is continuous. The first term goes to as t ^ by the convergence of Ht 
to H on £'[0, 1], and the second term also tends to by the continuity of H, 
because limt_fO ut ~ u. □ 

Proposition 7.2 (Hadamard differentiability of V and TZ). Under Condition C.l, 

(i) V is Hadamard- differentiable at (Gq^Gi), tangentially to C[0, 1]^, with 
derivative 

Vgo.Gi : [Ho, Hi) ^ ttoTg (ttoHo + (1 - ^o)i^i) + 7roi/o(r(G)) 

(a) TZ is Hadamard- differentiable at G, tangentially to C[0, 1] , with derivative 

■Rg-H^ H{T*)+g{T*)fG{H) 

Proof of Proposition 7.2. (i) Let [Hq, Hi) € C[0, 1]^, and (fl"o,t, fl"i,t)t>o be a 
family of functions of D[0, 1]^ that converges to {{Hq,Hi), ||.|joo) ast ^ 0. 
Let H = ttoHq + (1 — no)Hi, and Ht = noHo^t + (1 ^ ti'o)-^!,*- We have 

V(Go + tHo^t, Gi + tHi^t) - V(Go, Gi) = - t*) + ^o^^fo.tW) 

where r* = T(G) and denotes T(G -\-tHt). By the Hadamard differen- 
tiability of T at G tangentially to G[0, 1], we have, as H — ttqHo + (1 — 
TTo)Hi is continuous at t*, 

t: -r* =t{fG{H) + o{l)'^ 

In order to conclude, we notice that 

lunHoArn^Hoir*) 

according to Lemma 7.1, which concludes the proof. 

(ii) Let H g G[0, 1], and Ht be a family of functions of £'[0, 1] that converges 
to7£ on (£)[0,l],||.||oo) as t 0. We have 

■R{G + tHt) = {G + tHt)T{G + tHt) 

= G{T{G + tHt))+tHt{T{G + tHt)) 

By the Hadamard differentiability of T at G tangentially to G[0, 1], we 
have 

r(G + tHt) = T(G) + t {Tg{H) + o(l)) 
so that applying Taylor's formula to G at T(G) yields 

G(r(G + tHt)) = G{T{G)) + t (fG{H) + o(l)) g(r(G)) + o{t) . 
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For the second term, Lemma 7.1 ensures that 

Inri i/t(T(G + tHt)) = H{T{G)) 

because T{G + tHt) converges to T(G) and Ht converges to H on (D[Q, 1], 
1 1. 1 loo). Finally, we have 

lim = H{t ) + .g(r )Tg{H) 

because t* = T(G), which concludes the proof. 

□ 

Theorem 7.3 (Asymptotic distribution of {t,v,p)). Under Condition C.l, 
where 

X= 1^ TTo J TgIZ) +7ro ^ij Zo(r*) + (l-^o) ^ M^*) , 

withlj = 7roZo + (l — 7ro)Zi , w/iere Zq and Zi are independent Gaussian processes 

such that Zq B and Zi B o Gi , and B is a standard Brownian bridge on 
[0,1]. 

Proof of Theorem 7.3. We note that 
where ^ : _D[0, 1]^ M'^ is the map defined by 



/T(7ro^^o + (l-7ro)^^i)' 
\TliTToFo + (1 - 7ro)i^i) , 



We have 



'i'(Go,Gl) = ( 7T0T* 

By the Hadamard differentiability of T at G = ttqGo + (1 — 7ro)Gi and that of 
V at (Go, Gi), 4" is Hadamard-differentiable at (Go, Gi) tangentially to G[0, 1]-^, 
with derivative 
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where H denotes TTo-ffo + (1 ~ t^o)Hi. Therefore Theorem 3.1 yields 



VrrI(*(Go,™,Gi,„0 - ^'(Go,Gi)) ^ ^'go.Gi (^o, Zi) 

According to Proposition 7.2, we have 

Vgo,Gi(Zo,Zi) = ^otG(Z)+7roZo(T*) 
nam = giT*)fGiZ)+Z{T*) 

with Z = TToZ + (1 — 7ro)Zi, so that 

X = ^-G^^Gi (Zo,Zi) 

rG(Z) + ^0 I 1 ^o{t*) + (1 - ^o) I Zi(r*) 




, which concludes the proof. □ 
Proof of Theorem 3.2. [i] is a direct consequence of Theorem 7.3: 

^{t{^.„,)-t*)) -tG(Z), 

For (m) and (im), we note that as r* > (by Condition C.l), t = T(G,„) is 
bounded away from if to is sufficiently large, with probability 1. Specifically, 
there exist io G (0; 1) and toq € N such that 

P(Vto > 7710,? > to) = 1 • 

Therefore, as Gm is non decreasing, and as Gm(to) — > G(io) > 0, the proportion 
p = Gm{T) of rejections by procedure T is bounded away from with probability 
1. Thus, P (i?m (r) > 0) = 1, where Rm {T) ~ 777Gm(r) is the number of rejections 
at threshold r. 

As a first consequence, as FDR(t) = p{t)P {R{t) > 0), we have FDR,„(r) = 
p(t) for a sufficiently large to, which proves (ii) as p(r) converges almost surely 
to p{t*). 

As a second consequence, letting 7 : M+ x M!j_ ^ M be defined by 7(3;, v) = ^1 
we have FDPj„(T(G„i)) = 7 (i^, p) = 7 (i^i p) for a sufficiently large to, with 
probability 1. 7 is differentiable on R+ x M!j_, with derivative 

In particular, j-noT* .G{T*){h, k) ~ g{t*) G(t*) ^) • '^^^ therefore derive 

the asymptotic distribution of FDP„i from Theorem 7.3 combined with the Delta 
method [27]. According to Theorem 7.3 we have 

^\ _f TTOT* \ \ /7roZo(T*) + ^otG(Z) 
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Hence, as FDPm(T(G,„)) = 7(9, p) (almost surely), and ^{ttqt* ,t* / a) = noa, 
the Delta method [27] yields 

(^FDP,„(r(G„)) - - X, 

where 

^ = ^(^o(Zo(r*)+rG(Z))-^(Z(r^)+5(r*)tG(Z))) 

G(t*) ^ T* G{t*)) G{t*)\ G{t*) ) ' 

ksTL = -K^l^Q + (1 - TTo) Zi and G = ttoGq + (1 - 7ro)Gi, and p* = we 
have 

Z(r'^) .Zo(r*) ,, Zi(r*) 
gF)^'^^^^ '^G^' 

so that 

Zo(r*) Z(r*) ^Zo(r*) Zi(r*) 

^ (1 - P ) 



G(t^) ^ ^ V Gi(t*), 
which concludes the proof because p(t) ~ and p(t) = v' ~ ' 



7.1.2. Proof of Proposition 3. 6 

Lemma 7.4 states the asymptotic equivalence between a multiple testing proce- 
dure defined as a threshold function and a slight modification of this procedure. 

Lemma 7.4. Let T be a threshold function, e ~ (£m)meN o-nd : -0(0, 1] 
[0, 1], such that 

eD[Q,l]X'f{F)^T{F + e^H) . 

Let Ai be the multiple testing procedure naturally associated with the sequence 
of thresholds 7^(Gm). If Condition C.l holds for T, and if Em — o (^"^^^ ; then 
A4 is asymptotically equivalent to T as m —i- +00. 



Proof of Lemma 7.4- The proof is based on the idea that, as Sjji — o 




and Gm converges at rate to G, a modification of T of the order of Em 
does not change the asymptotic distribution of the associated FDP, because T 
is Hadamard-differentiable. For the sake of simplicity in notation, we prove only 
that 

V^(T^'^(G„o-r(G„o) ^0. 

Indeed, as the associated FDP is a Hadamard-differentiable function of the 
empirical distribution functions under the null and alternative hypotheses Go,m 
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and Gi^m, the arguments developed below can be transposed (but with much 
more cumbersome notation) to prove that 

V^(FDP,„(r-'^^(G„0) - FDP,„(r(G,„))) 4 0. 

Let Zm = ^/m(Gm — G) . According to Donsker's invariance principle (The- 
orem 3.1), Zm converges in distribution on [0,1] to a Gaussian process with 
continuous sample paths. For Z G -D[0, 1], let 



(/)™(Z) = ( T''" [ G + -^z] - t(g+ -^Z 



m 



We have 



According to Condition C.l, T is Hadamard-difFcrcntiable at G tangcntially 
to C[0, 1]. Therefore, for any sequence of I?[0, 1] that converges to Z G 

C[0, 1], (t (g + -^Zrn) - T{G)) converges to faiZ). As £„, = o (^) , 

^ (t^'" (^G + ^^m) - T{G)^ also converges to faiZ). 

Thus, <t>m{.Zm) converges to for any sequence Z,„ of _D[0, 1] that converges 
to Z G C[0, 1]. Therefore, according to the Extended Continuous Mapping The- 
orem [27, Theorem 18.11], (/)m(Zm) converges in distribution (hence also in prob- 
ability) to 0. □ 

Proof of Proposition 3.6. For m £ N, let : ^[0, 1] [0, 1] be defined by 
Let T* = T{G), ? = r(G™), ?,n = %n{Grn) and ?^ = Write 

where Go.m and Gm are non decreasing functions, so that 

7roGo,,n (r^) 7ro(Go.„, (?) 

(f) (TfJ 

because < f,„ < t. Therefore, 

TTO (Go,™ (Tf„) - Go,m (f)^ 



< FDP„,(f„0 - FDP,„(?) 

Gr„ (r) 
TTO (Go,™ (r) - Go,m (Tf„) ) 

^ ^ ^-J^^ - (FDP™(?) - FDP,„(rf„)) 

Gm (7"mj 
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As 7^f; < %n < T for any m S N, Lemma 7.4 ensures that ^/m — r) and 
y/rri{f^ — r) converge to in probability. Therefore, as G,„ (t) and G,„ (t^^J 
converge in probabihty to G (T{G)) as m +oo, we have 



T^oVm (Go,m (Tf„) - Go,m (t) 



(r) 

and 



4o 



^0, 



which concludes the proof because ^/m (FDP,„(t) — FDP,„(rj|;j)) also converges 
in probability to (according to Lemma 7.4). □ 



7.2. Asymptotic FDP; specific threshold functions 

Wc now apply the results of section 7.1 to threshold functions of the form 

T(F) =U{F,A{F)), 

with 

U{F,a) = sup{u e [0,1], F{u) > r„(u)}. 

In this section, we will use the notation r : {a,u) i—* ra(u) whenever the de- 
pendence of ra in a is of importance. We begin by giving sufficient conditions for 
the regularity of W and A under which Condition C.l holds (section 7.2.1, Propo- 
sition 7.5). Then we provide sufScient conditions for U to be regular enough to 
be consistent with hypotheses («) to {Hi) of Proposition 7.5 (section 7.2.2). Fi- 
nally we derive the asymptotic distribution of the corresponding False Discovery 
Proportion (section 7.2.3). 



7.2.1. Hadamard differentiability of T 

Proposition 7.5 (Hadamard differentiability of T). Let C[0, 1] be the set of 

continuous functions of D[0,l]. Suppose that 

(i) U is Hadamard-differentiable with respect to its first variable at {G,a), 
tangentially to C[0, 1], for any a in a neighborhood of A{G); its derivative 
will be denoted by Vp^G.a,' 
(ii) SIfIAg,. is continuous atA{G); 

(Hi) hi is differentiate with respect to its second variable; its derivative will be 

denoted by VaU(GMG)); 
(iv) A is Hadamard-differentiable at G tangentially to C[0,1]; its derivative 

will be denoted by Ag- 
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Then, T is Hadamard-differentiable at G tangentially to G[0, 1], with deriva- 
tive Tq, defined for any H Cz D[0, 1] by 

TaiH) = VfU(^g,a{g)){H) + Ag{H)V^U^g,a{g}) ■ 

Proof of Proposition 7.5. Let H G C[0, 1], and Ht be a family of functions of 
D[0, 1] that converges to H on {D[0, 1], ||.||oo) as t ^ 0. 

As A is continuous at G (by (iv)), A{G + tHt) lies in a neighborhood of 
A{G) for small t > 0, and U is Hadamard-differentiable with respect to its first 
variable at {G,A{G + tHt)) by (i). We therefore have 

T{G + tHt) = U{G + tHt,A{G + tHt)) 

= U{G,A{G + tHt)) + t^FUG^A(G+tH,) {H){1 + 0(1)) 

= U{G,A{G + tHt)) + fUg^a(g){H){1 + o(l)) 
by the continuity oi\J pl^G.- at .4(G) {ii). Then, combining (iii) and {iv) yields 

U{G,A{G + tHt)) = u(G,A{G) + tAG{H){l + o{l))^ 

= U{G,A (G)) + tAG{H)VMiG.A[G)) (1 + o(l)) 

= r(G)+UG(i?)V„Zi(G,^(G)) (1 + 0(1)) 

so that 

lim '^iG + tHt)-r{G) ^ ^j^u^^ ^(G)mAG{H)WM(G,A(G)). 
which concludes the proof. □ 



7.2.2. Regularity ofU 

The crucial point for proving the desired regularity of hi is its Hadamard dif- 
ferentiability with respect to its first variable at (G, a), tangentially to G[0, 1], 
for a in a neighborhood of .4(G). Lemma 7.6 is a straightforward analytical 
translation of Conditions C.2 and C.3. 

Lemma 7.6. Under Conditions C.2 and C.3, the unique interior right crossing 
point T* between r^ and G is positive. If r is G^ on (0,1] x [0,1], there exists 
a neighborhood V = A x U of {A{G),t*) such that for any (a,x) £ V, ipG,a '■ 
u I— !■ ra{u) — G{u) is locally inversible around U {G , a) , with ipG. aix) > 0. 

We begin by proving the continuity of U at (G, a) for a in a neighborhood of 
.4(G). We then (Proposition 7.11) provide the sufficient conditions for conditions 
(i), {ii), and {Hi) of Proposition 7.5 to hold. 

Lemma 7.7. For any F e D[0, 1], and a G [0, 1] such that ra is continuous, 
one of the following two assertions holds: 

(i) F{U{F,a))=r^{U{F,a)) 
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(li) F{U{F,a)) < ra{U{F,a)) < F{U{F,a)-) 

Proof of Lemma 7.7. According to the definition of U{F,a), F{u) < ra{u) for 
any u > U{F,a). As F is right-continuous and Tq, is continuous, we have 
F(U{F, a)) < raiU{F, a)). Therefore, either (i) holds, or F{U{F, a)) < raiU{F, a)) 
In the second case, according to the definition oiU{F, a), there is a non decreas- 
ing sequence (tin) that converges to U{F, a) such that F(u„) > rct(w„). As is 
continuous and F is left-continuous, we have F{U{F, a)^) > ra{U{F, a)), which 
proves (m). □ 

Proposition 7.8. Let F G D[0, 1] be non decreasing, and a = ■A{F). If ra is 
continuous, then 

F{U{F,a)) ^rc{U{F,a)). 

Proof of Proposition 7.8. Let us consider the two assertions of Lemma 7.7: as 
F is non decreasing, (m) can be reduced to (i). □ 

Proposition 7.9. Let r be continuous on (0, 1] x [0, 1]. Let F G D[Q, 1] be non 
decreasing, and a G (0, 1]. Let Ft be a sequence of functions of D[0, 1] .such that 
{Ft)t>o converges to F on (Z?[0, 1], ||.||oo) as t 0, and at a as t 0. 
Denote by 'ipF,a the function defined on [0, 1] by 

\fu G [0, 1],V'-F,q(u) = ra{u) - F{u) . 

Then, 

lim ^/-Ko {U{Ft,at)) = V'F,a [U{F, a)) . 

Proof of Proposition 7.9. For each fixed t G [0, 1], one of the following two as- 
sertions holds according to Lemma 7.7: 

(i) Ft{U{Ft,at))^r^Sl^{Ft,at)) 

(ii) Ft{U{Ft,at)) < r^SU{Ft,at)) < Ft{U{Ft,at)-). 

If [a) holds, then, as F is non decreasing we have 

{F-Ft)(U{Ft,at)-) < FiUiFt,at))-r^^{U{Ft,at)) < {F - Ft){U{Ft,at)) . 

If (z) holds, then F{U{Ft,at))~ro,,mFt,at)) - F{U{Ft,c^t))-Ft{U{Ft,at)). 
In either case, we have \F{U{Ft,at)) — rat{U{Ft,at))\ < \\F — i^t||, which tends 
to as i ^ 0. As r is continuous on (0, 1] x [0, 1], converges uniformly to Tq, 
on the compact [q:/2, 1], and we have 

\im F{U {Ft, at)) ' r^{U {Ft, at)) = , 

which concludes the proof as ipF,a {^{Fj a)) = 0. □ 

Corollary 7.10 (Continuity of U). Let r be on (0,1] x [0,1]. According 
to Conditions C.2 and C.3, there is a neighborhood A of A{G) such that U is 
continuous at {G,a) for any a G A. 
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Proposition 7.11 (Differentiability oiU). Let us assume that r is on (0, 1] x 
[0,1]. Under Conditions C.2 and C.3, 

(i) lA is Hadamard-differentiable with respect to its first variable at (G, a), tan- 
gentially to C[0, 1] for any a in a neighborhood A of A{G), with derivative 
Vf^G.o defined for any H G C[0, 1] by 

V (m H{U{G,a)) 

(a) Vpl^G,. is continuous at A{G) on A. 

(Hi) hi is differentiable with respect to its second variable, with derivative 

dr 



§^iA{G),r*)) 



where t* ^U{G,A{G)). 

Corollary 7.12. // we also assume that A is Hadamard-differentiable at G, 
tangentially to C[0, 1], then T is Hadamard-differentiable at G, tangentially to 
C[0, 1], with derivative defined for any H G C[0, 1] by 

j- ^ H{r*)^^{A{G),r*))AG{H) 
t{AG),t)-9{t) 

Proof of Proposition 7.11. Let t* = U{G,A{G)). Throughout the proof, V = 
Ax U denotes the neighborhood of {A{G),t*) defined in Lemma 7.6. 

(i) Let a e A. Let H e C[0, 1], and Ht be a family of functions of D[0, 1] 
that converges to H on (D[0, 1], || .||oo) as t ^ 0. Let v = U{G,a) and 
vt = W(Gt, a), with Gt = G+tHt. By the continuity of U (Corollary 7.10), 
Vt ^ V ds t ^ Q. Therefore, applying Taylor's formula to ipca ■ u i— *■ 
Taiu) — G{u) yields 

i^G.aivt) - IpG.aiv) [vt - v)lpG.aiv) (1 + o(l)) . 

As a e A, we have ^pG,a{v) = ^^{'^i > 0- Therefore, as V'G,a(^) = 

according to Proposition 7.8, 



V'G,a(«t) 



SO that it is sufficient to prove that limt_»o i^G,a{vt) /t = H{v). The behav- 
ior oiipG,a{,vt) = ra{vt) — G(vt) can be derived using the same argument as 
in the proof of proposition 7.9; Lemma 7.7, we have either Gt{vt) = ra{vt) 
or Gt{vt) < ra{vt) < Gt{v:[). In the first case, ra{vt) - G(vt) = {Gt - 
G){vt) = tHt{vt), and limt_o ''"'"'^^'^'""^ = H{t) according to Lemma 7.1. 
In the second case, we have 

{Gt - G){vt) < rM - G{vt) < {Gt ~ G){v^) 
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as G is non decreasing, that is, tHt{vt) < V'G,a(i't) < tHt{vf ) . Therefore, 
we have 

Ht{vt) - H{vt) < t^^^ - H{vt) < Ht{vi) - H{vt) . 

As H is continuous, H{vt) — H{v^), and the upper and lower bounds 
converge to according to Lemma 7.1, and (i) is proved. 

(ii) is a consequence of the continuity oiU (with respect to its second variable), 
that of g and that of V„r with respect to its first variable. 

(iii) Let a = A{G). Let at be a sequence of points of (0, 1] that converges to 
a. Let V = U{G,a) and Vt — U{G,at)- By the continuity of U (Propo- 
sition 7.9), ft — > u as t ^ 0. Therefore, applying Taylor's formula to 
ipG,a '■ u 1-^ ra{u) — G{u) yields 

i^Caivt) - 1pG,a{v) (w* - w)'0G,q(w) (1 + o(l)) . 

We have 'ipG,a{v) ~ and ipG.a{vt) = ^{a, vt)—r{at,vt) by Proposition 7.8. 
As r is in a neighborhood of {a,v), we have, according to Taylor's 
formula, 

dr 

r{at,vt) - r{a,vt) = (at - a) — (a, ?;) (1 + o(l)) . 
t— ►o oa 

As a = ^ and v ~ U{G,a), (a,u) G V. Therefore i^ca {v) = ^{a,v) — 
g{v) > according to Lemma 7.6. Finally, we have 

^.^ r{a,vt) -r{at,vt) _ §^{(^,v) 



at -a »L(^a,v)-giv)' 
which concludes the proof. □ 



7.2.3. Asymptotic FDP 

Theorem 7.13 (Asymptotic distribution of FDPm). Let be a rejection curve 
such that r is on (0,1] x [0,1], and A a level function. Let us denote by 
T : F i—f L{{F,A{F)) the associated threshold function, where U{F, a) = supjit € 
[0,l],F(w)>r„(w)}. 

Under Conditions C.2 and C.3, if A is Hadamard-differentiable at G tan- 
gentially to C[0, 1], then 



^r*) ~ §^{A{G),t*))Ag{^) 



(ii) 



^iA{G),t) ~ g{t) 
V^(FDP,„(r(G™))-p*) -^X, 
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where p* ~ a^r*) pFDR achieved by procedure T , p(t) = ^1 — ^^j- 

and 

X = p*(l - p*C(r*))?^^ + p*(l - p*)C(r*)^4^T + P(^*)C(r'^MG(Z) , 

^^^/^ cw ^ -f=^|)^, = ^."(g^tgi-;;,) : z = -oZo + (1 - 

7ro)Zi, where Zq a77.d Zi are independent Gaussian processes such that 
Zq == B and Zi *■= B o Gi, and B is a standard Brownian bridge on [0, 1]. 

Proof of Theorem 7.13. Under these assumptions, Condition C.l holds for T 
according to CoroUary 7.12, with 

f (jj. ^ H{r*)^l^{A{G),T*))AG{H) 
^{A{G),t)~g{t) 

Therefore, Theorem 3.2 yields y/m (r(G™) - t*)) Tg(Z), and 

(^FDP™(T(G™)) - ^ X , 

withX = p*(l-p*) (^4^ - +p(T'^)rG(Z) and Z = ^o^o + (1 - 7ro)Zi, 

where Zq and Zi arc independent Gaussian processes such that Zo == B and 

Zi ="* B o Gi, and B is a standard Brownian bridge on [0, 1]. 
Letting 5{t) = , we have 

rG(Z) = 5(r*) (^Z(r*) - ^(^(G),r*)MG(i?) 
As p(t) = p(f)(i-^), wehave 

(G{t*) \ Z(r^) 



p(r*)^(r'^)Z(r*) = p*^^(^') 



with ^ = p-Min + (i_p*)|i(ia. Hence letting C(t) = l-5{t) - 
we have 



X = p'^(l - pX(r*))^^^ - p^(l - p*)C(r'^)^4^ + p(r'^)^(r'^)^G(Z) , 
where ^{t) — ar- \\ (.\ • This concludes the proof since may be written 
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7. 3. Limit distribution for procedures under consideration 
7.3.1. One-stage procedures 

In this section A{G) is fixed. Therefore, only the dependence of u is of 
importance. In order to lighten the notation we let 

dr 

Theorem 7.14 (Asymptotic FDP for one-stage procedures). Let T : F ^ 
L({F,a) a one-stage procedure such that r^ is continuous on [0, 1], and in a 
neighborhood of t* ~T{G). Under Condition C.2 and C.3, 



m(r(G„o-T*)) 



(ii) 



with 



mfFDP™(T(G,„))-pi 



X ^ p*(l - p*C(r*))^ - p*(l - P^))C(r*)^ , 

T* Gl[T*) 

where p* = C{t) = ^^^yj^z^ ' '^'^^Z = ttoZq + (1 - 7ro)Zi, where Zq 

and Zi are independent Gaussian processes such that Zq B and Zi MoGi, 
and M is a standard Brownian bridge on [0, 1]. 

Proof of Theorem 7.14- As T is a one-stage procedure, we have A ~ a. There- 
fore, the assumptions for Theorem 7.13 hold, with f = 0. According to Proposi- 
tion 7.8, G{t*) = ra{T*)\ therefore p* = .i^"(t*) ■ ^^i^h concludes the proof. □ 

Proof of Theorem 4.2 (BH95). Uniqueness Condition C.3 always holds because 
Tq is linear, and Condition C.2 holds because it corresponds to Condition C.4. 
Therefore, Theorem 7.14 can be applied, and we have C,{t*) = since r'Q,(r*) = 
l/a = ra{T*)/T* , and p(t*) = Troa. Hence, 



(t(G™) - T*) 



l/a- g{T*) 



and ^ 
(^FDR,„(T(G„i)) - ttoo^ naa-^ — , 

which concludes the proof because VarZo(r*) — t* {1 — t*). □ 
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Proof of Theorem 4-7 (FDR08). The uniqueness Condition C.3, and existence 
Conditions C.4 and C.7 ensure that there is a unique interior right crossing 
point T* between fa and G, which satisfies t* < k. Condition C.6 guarantees 
that T* is also the only right crossing point between and G. Thus, [0, k] is 
a neighborhood of t* in which coincides with fa and is C^, with fa{u) = 
,^a+(i-a)u)'' - Therefore, Theorem 7.14 yields ^{fDP^{T{G^)) - p*) ^ X, 
with 

X = p (1 - p C(t ))^; P (1 - P )C(t ' 

where p^ = = 7ro(a + (1 - a)T*) and C(t*) = - ^^p^/^p^^ly^ • Letting 

l-G(t) 

we have G(T*)/r'^ = ^(T*)/a, and /a(T*) = a(/„(T*)/T*)2 = W(T*)Va, so 
that p* = ano /ttq (t*) , and 



C(r*) 



G{t*)/t* - fair*) 

faiT*)~g{T*) 

T^{T*)la-T^{T*f /a 
-W^{T*y/a - g{T*) 

—, W(T*)/a 



7ro(T*)2/a - gr(r*) ' 

which concludes the proof. □ 

Proof of Theorem 4. 10 (BR08(A)). The uniqueness Condition C'i. and existence 
Conditions C.8 and C.9 ensure that there is a unique interior right crossing point 
T* between and G, which satisfies t* < A. Thus [0, A] is a neighborhood of 
T* in which 6^ is G^, with ba{u) ~ (^a{i-\)+uf ' Therefore, Theorem 7.14 yields 
V^(FDP™(r(G,„)) - P*) ^, with 

X = p (1 - p C(t ))^; P (1 - P )C(t ' 

where p^ = ^^a^ = 7ro(a + (1 - a)T*) and C(t^) = -^f^;Tp^77p- We have 
/„(r^) = a(l - A)(5„(T*)/r*)2 = G(t*)(1 - G(t*))/t*, so that 



C(r*) 



_G(t;^)/t*_U^ 

G(r-)/r-(l-(l-G(r*))) 
G{t*){1 - G{t''))/t* - g{T*) 

G(t*)Vt* 

~ G(t*)(1 - G{t*))/t* - g{T*) ' 

which concludes the proof. □ 
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7.3.2. Two-stage adaptive procedures 

Proof of Theorem 4.12. As pointed out in section 4.3, Condition C.3 always 
holds because is linear, and Condition C.2 holds as soon as A{G) > a*. 
Therefore, Theorem 7.13 yields VTO(FDP™(r((G„)) - p*) X, with p* = 
■KoA{G), and 

X = p^l - C(r*)p*)^^ - p^(l - P*)C(r*)§|^ + pir*nr^)AGiZ) , 

where p(r*) = ^ {^~9{r*)), CW "^giSSif ' ^^^^ ^^'^ = 
fir- . Simes' line is defined by Tq, : w i— > u/a. Therefore, we have 

|^(^(G),t) = ^ and §^{A{G),t) = and G(r*) accord- 

ing to Proposition 7.8. We have C(t*) = 0, ^(r*) = and p(t*) = 

A(G) S*."^ ^ 

P'^^^rp- (^A(G) ^ 9i'''*)j ' which concludes the proof. □ 

Sto02 procedure. The following Proposition establishes the Hadamard dif- 
ferentiability of the level function of procedure Sto02. The proof is immediate. 

Proposition 7.15. For F e ^[0, 1], let 

. 1 - A 

AiF) 

ndition 

tangentially to C[0, 1], with derivative 



i-F{xy 

where a G [0,1]. Under Condition C.ll, A is Hadamard- differentiable at G, 



Aaim^AiG)- ^^^^ 



1 - G(A) ■ 

Proposition 7.16. Let X £ [0, 1) such that Conditions C.9 and C.ll hold. Then 
procedures Sto02(A) and STS04(A) are asymptotically equivalent. 

Proof of Proposition 7.16. Let A e [0, 1). According to Condition C.9, we have 
^Sto02(A) ^ ^ Therefore, procedure Sto02 is asymptotically equivalent to the 
same procedure truncated at A, that is, the procedure with threshold function 
defined for F G D[0,1] by 



u 1 — A 
sup { u e [0, A],F(it) > -- 



a 1 -F(A)} ' 

We thus work with this truncated version for the remainder of the proof. By def- 
inition, the rejection curve of procedure STS04 is larger than that of procedure 
Sto02. Therefore, we have r™™'*'^'(F) < r'^^°on>^) [F) for any F £ D[0,1\. 
With the same argument we also have rSt°02(A) (F - ^) < T^^^'^'^^^\f) for 
any F G fp, 1]. As we have assumed that Condition C.ll holds, Condition C.l 
holds for T according to Proposition 7.15, and the result follows from Proposi- 
tion 3.6. □ 
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Proof of Theorem 4-15. According to Proposition 7.15, and because A{G) > a*, 
Theorem 4.12 ensures that 

where 

Denoting 7f5'(A) = 7f5''^(A); this may be written as 

_/p^p ^0 ^ 7T0 fZoir*) , Z(A) 
m FDP„(T(Gm)) - ... a | . a 



7f^(A) ; W(A) V ^* l-G(A) 

For the calculation of variance, it suffices to note that VarZo(T*) = r*(l — r*) 
and 

E[Zo(t*)Z(A)] = ^oIE[Zo(t*)Zo(A)] 
= TTo (t* a a — r*A) , 

which concludes the proof since Zo and Z are centered. □ 

Procedure BKY06(A)[2]. According to Proposition 7.8, we have /?)) = 

U{F,(3)/P for any F e D[0, 1], so that the level function of procedure BKY06 
may be written as 

1-Z^(F,A)/A- 

We now prove the Hadamard differentiability of the level function of proce- 
dure BKY06(A) under Condition C.12. 

Proposition 7.17. For A G [0, 1) and F £ D[0, 1], let A{F) = j^j^j^- Un- 
der Condition C.12, A is Hadamard- differentiable at G, tangentially to C[0, 1], 
with derivative 

Ag{H)- H{U{G,X)) 



a(l - A) l/(a(l - A)) - giU{G, A)) 



Proof of Proposition 7.17. As Condition C.12 holds, Condition C.4 holds for the 
BH95 procedure at level A: U is Hadamard-differentiable with respect to its first 
variable at (G, A), tangentially to C[0, 1], with derivative Vf1^g,\ defined for 
any H e C[0, 1] by 

l^{\M[G,X))-g{U{G,\)) 

As the rejection curve of U is Simes' line, we have ^{\,U{G,\)) ~ j. As 
A{F) = a(j){U{F, A)), where cf) : x ^ {1 - A)/(l - x/X) is derivable for x ^ X, 
with (l)'{x) — the result follows from the chain rule. □ 
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Proof of Theorem 4-20. As Condition C.12 holds, this is a direct consequence 
of Proposition 7.17 and Theorem 4.12. □ 

7.4- Connections between one-stage and two-stage adaptive 
procedures 

Proof of Theorem 5.2. As we have assumed that existence Condition C.2 and 
uniqueness Condition C.3 hold for procedure T, t* = T(G) is the only point in 
(0,1) such that G{t*) = Ca{G,T*). Similarly, as existence Condition C.2 holds 
for procedure % for any t € (0, 1), r(<) = 7t(G) is the only point in (0, 1) such 
that G{T(t))/T{t) = Ca{G,t)/t. Therefore, we have 

t<T* ^ Git)>Ca{G,t) 
^ G(t) ^ c^iG,t) 

t ~ t 
^ G(t) ^ GjTjt)) 

t - T{t) 

t < r(t) 

as u i-^ G(u)/u, is non increasing (due to the concavity of G). As w i-^ Cq(G, u)/u 
is non increasing (Condition C.13), we have 

t<r* ^ ^c.{G,T*) ^ c^{G,t) 

T* ~ t 

^ G{t*) ^ GjTjt)) 

T* - Tjt) 
^ Tjt) < , 

and ji) is proved. Let A G (0, 1). If A < r*, then by (i), the sequence (t„) is non 
decreasing, and smaller than t*. It therefore converges to a limit i € [A, t"^], 
such that rja) = £, that is, Gj£) = CajG,£). The uniqueness Condition C.3 
ensures that £ = t* . Conversely, if A > t*, then, by ji), the sequence jtn) is non 
increasing, greater than t*, and thus converges to ^ € [t* , A] such that Tj£) = £, 
and we also have £ = t*. □ 

Sto02 and FDR08. 

Proof of Theorem 5.4. As existence Condition C.4 holds, existence Condition C.ll 
also holds for procedure Sto02(i), for any t G (0,1). Therefore, Theorem 4.15 
ensures that the asymptotic threshold rjt) of procedure Sto02(t) is positive, and 
satisfies Gjrjt)) = ^^tTq (t), where ttq ju) = 

As uniqueness Condition C.3 and existence Conditions C.4 and C.7 hold, 
Theorem 4.7 ensures that the asymptotic threshold t* of procedure FDR08 
satisfies t* e (0, k), and satisfies Gjr*) = /q(t*), where fa : u i-^ u/ ja + jl — 
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a)u) is the rejection curve of the FDR08 procedure. For any fixed A G (i A k, 1), 
the FDR08(A) procedure defined by the capped threshold function 

Ta {F) = sup {m e [0, A] , > f^u} 

also has asymptotic threshold t* according to Proposition 4.6, as A > k. For 
F e D[0, f] and u e [0, A], let 

a 1 — u 

As G is concave, u ^ "'""^^"^ is non increasing, so that Cq, fulfills the re- 
quirements of Condition C.f3. Therefore, as F(u) > faiu) may be written 
as F{u) > Ca{F,u), the result follows from the application of Theorem 5.2 to 
procedures FDR08(A) and Sto02(t). □ 

BKY06(A) and BR08(A). 

Proof of Theorem 5.6. As uniqueness Condition C.3 and existence Conditions C.^ 
and C.9 hold, Theorem 4.10 ensures that the asymptotic threshold r* of proce- 
dure BR08(A) is the unique point in (0, A) such that G(r*) = a(i^\)+T* ' because 
the rejection curve 6^ of the BR08 procedure equals ^^^^ J^^^^ for u < \. 

Existence Condition C.8 also ensures that r(t) exists for any t < A. For 
F e ^[0, 1] and u e [0,A], let 

Ca(-F,u) = - ^ ^^f^ . 

a 1 — A 

As 1 — G is non increasing, fulfills the requirements of Condition C.13. There- 
fore, as Fill) > ba{u) may be written as F{u) > Ca{F, u), the result follows from 
the application of Theorem 5.2 to procedures BR08(A) and BKY06(A). □ 

Proof of Corollary 5. 7. According to the definition of ma as the asymptotic 
threshold of the BH95 procedure at level A, the asymptotic threshold t* of 
procedure BR08(A) satisfies r* > u\ if and only if G{t*) < r*. According to the 
definition of the rejection curve 6^ of the BR08(A) procedure, this is equivalent 
to T*/{a{l - A) + T*) < T*/X, that is, to r* > A - ^(l - A). □ 
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Notation used in the paper 



Hi 
m 

mo(m) 

TTo = mo{m)/m 

{Pi)ie{l...m.} 

M 

FDP= -5%^ 



T 

ra = r(a, ■) 

A 

U 

p(f) = pFDR(t) 
m = ^(PFDR)W 

Go : w ul[o,i](?t) 
Gi 

Si = G'l 

G = 7roGo + (l-7ro)Gi 
P = TTO + (1 - 7ro)3l 



mo(m) 



Table 2 

Notation used throughout the paper 

null hypothesis 
alternative hypothesis 
number of tested hypotheses 
number of tested hypotheses 
proportion of true null hypotheses 
associated p- values 

Multiple Testing Procedure (MTP), cf Definition 2.1 
number of rejections among m hypotheses for a given 
MTP 

number of false rejections among m hypotheses for a 
given MTP 

FDP attained by a given MTP 

FDR attained by a given MTP 

threshold function of a MTP, cf. Definition 2.2 

rejection curve of a MTP 

level function of a MTP 

{F, a) ^ sup{M G [0, 1], > rc«(u)} 

positive pFDR attained at t 

distribution function of p-values under Ho (Uniform) 
distribution function of p-values under Hi {C^ , con- 
cave) 

density of p-values under the alternative hypothesis 



empirical distribution function of p-values under Ho 
(unobservable) 



m — rriQ (m) 
B 

Zo B 

Zi = B o Gi 

Z — TToZo + (1 — vr())Zi 

r*=T(G) 

a* = lim„^o ^ 



empirical distribution function of p-values under Hi 
(unobservable) 

empirical distribution function of p-values (observable) 
standard Brownian bridge on [0, 1] 

limit in distribution of ^/m{Go,m — Go) 
limit in distribution of ^/m{Gi.,n — Gi) 
limit in distribution of y'rn{Gm — G) 
asymptotic threshold of the MTP with threshold func- 
tion T 

critical value of the BH95 procedure, as defined by [5] 
procedure A is more powerful than procedure B 
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